Method and apparatus for controlling head-up display based on eye tracking status

ABSTRACT

A method and apparatus for controlling a head-up display (HUD) considering an eye tracking status are provided. The method includes identifying an eye tracking status based on a result of an eye tracking, and identifying a rendering mode for an HUD image to be one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking status.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Korean Patent Application No.10-2020-0106554 filed on Aug. 24, 2020, and Korean Patent ApplicationNo. 10-2020-0130402 filed on Oct. 8, 2020, in the Korean IntellectualProperty Office, the disclosures of which are incorporated by referenceherein in their entirety.

BACKGROUND 1. Field

Methods and apparatuses consistent with example embodiments relate to amethod and apparatus for controlling a head-up display (HUD) based on aneye tracking status.

2. Description of the Related Art

A head-up display (HUD) system generates a virtual image in front of adriver of a vehicle and provides a variety of information to the driverby displaying the information in the virtual image. The informationprovided to the driver may include, for example, navigation informationand dashboard information such as a vehicle velocity, a fuel level, andan engine revolution per minute (RPM). The driver may more easilyrecognize the information displayed in front without turning his or hergaze during driving, and thus, driving safety may improve. In additionto the navigation information and the dashboard information, the HUDsystem may also provide the driver with, for example, a lane indicator,a construction indicator, an accident indicator, or a pedestriandetection indicator using augmented reality (AR), to assist with drivingwhen a field of view is poor and/or inadequate.

SUMMARY

One or more example embodiments may address at least the above problemsand/or disadvantages and other disadvantages not described above. Also,the example embodiments are not required to overcome the disadvantagesdescribed above, and an example embodiment may not overcome any of theproblems described above.

According to an aspect of the disclosure, there is provided a method ofcontrolling a head-up display (HUD), the method comprising: performingeye tracking of an eye of a user in a captured image; identifying an eyetracking status based on a result of the eye tracking; identifying arendering mode for an HUD image to be one of a two-dimensional (2D)rendering mode and a three-dimensional (3D) rendering mode based on theeye tracking status; and rendering the HUD image in the identifiedrendering mode.

The identifying the eye tracking status may comprise classifying the eyetracking status as one of a stable status and an unstable status basedon whether eye coordinates are present in the result of the eye trackingor based on a rate of change of the eye coordinates.

The identifying the rendering mode may comprise: identifying therendering mode to be the 3D rendering mode based on the eye trackingstatus being classified as the stable status; and identifying therendering mode to be the 2D rendering mode based on the eye trackingstatus being classified as the unstable status.

The eye tracking status may be classified as the stable status based onthe eye coordinates being included in the result of the eye tracking anda speed of change of the eye coordinates is less than a reference value.

The reference value may correspond to a system processing rate.

The eye tracking status may be classified as the unstable status basedon the eye coordinates being included in the result of the eye trackingand a speed of change of the eye coordinates is greater than a referencevalue, or based on the eye coordinates not being included in the resultof the eye tracking.

The HUD image may be rendered based on a first source image for a firstviewpoint and a second source image for a second viewpoint.

Based on the identified rendering mode being the 2D rendering, therendering the HUD image mat comprise rendering the HUD image by settingthe first viewpoint and the second viewpoint equally as a singleviewpoint.

The rendering the HUD image may comprise: setting, based on the resultof the eye tracking including current eye coordinates of both eyes and aspeed of change of the current eye coordinates being greater than areference value, the first viewpoint and the second viewpoint equally asa center viewpoint of the current eye coordinates; and setting, based onthe result of the eye tracking not including the current eyecoordinates, the first viewpoint and the second viewpoint equally as acenter viewpoint of previous eye coordinates.

Based on the identified rendering mode being the 3D rendering mode, therendering the HUD image may comprise rendering the HUD image by settingthe first viewpoint and the second viewpoint as different viewpoints.

Based on the rendering mode being identified, the rendering mode may beswitched from the 3D rendering mode to the 2D rendering mode or from the2D rendering mode to the 3D rendering mode during a buffer timecorresponding to a plurality of frames.

The HUD image may be rendered based on a first source image for a firstviewpoint and a second source image for a second viewpoint, and wherein,based on the rendering mode being switched from the 2D rendering mode tothe 3D rendering mode, the rendering the HUD image comprises renderingthe HUD image while gradually changing the first viewpoint and thesecond viewpoint to a single viewpoint used in the 2D rendering modeover the buffer time.

According to another aspect of the disclosure, there is provided anon-transitory computer-readable storage medium storing instructionsthat, when executed by a processor, cause the processor to perform themethod.

According to another aspect of the disclosure, there is provided anapparatus for controlling a head-up display (HUD), the apparatuscomprising: a memory configured to store one or more instructions; and aprocessor configured to execute the one or more instructions to: performeye tracking of an eye of a user in a captured image, identify an eyetracking status based on a result of the eye tracking, identify arendering mode for an HUD image to be one of a two-dimensional (2D)rendering mode and a three-dimensional (3D) rendering mode based on theeye tracking status, and render the HUD image in the identifiedrendering mode.

The processor may be further configured to classify the eye trackingstatus as one of a stable status and an unstable status based on whethereye coordinates are present in the result of the eye tracking or basedon a rate of change of the eye coordinates.

The processor may be further configured to: identify the rendering modeto be the 3D rendering mode based on the eye tracking status beingclassified as the stable status; and identify the rendering mode to bethe 2D rendering mode based on the eye tracking status being classifiedas the unstable status.

Based on the rendering mode being identified, the rendering mode may beswitched from the 3D rendering mode to the 2D rendering mode or from the2D rendering mode to the 3D rendering mode during a buffer timecorresponding to a plurality of frames.

According to another aspect of the disclosure, there is provided ahead-up display (HUD) device comprising: an eye tracking cameraconfigured to capture an image including a user; a processor configuredto perform eye tracking on the captured image, identify an eye trackingstatus based on a result of the eye tracking, identify a rendering modefor an HUD image to be one of a two-dimensional (2D) rendering mode anda three-dimensional (3D) rendering mode based on the eye trackingstatus, and render the HUD image in the identified rendering mode; and adisplay device configured to provide the HUD image to the user usingaugmented reality (AR).

The processor may be further configured to classify the eye trackingstatus as one of a stable status and an unstable status based on whethereye coordinates are present in the result of the eye tracking and basedon a rate of change of the eye coordinates.

The processor may be further configured to, based on the rendering modebeing identified, switch the rendering mode from the 3D rendering modeto the 2D rendering mode or from the 2D rendering mode to the 3Drendering mode during a buffer time corresponding to a plurality offrames.

According to another aspect of the disclosure, there is provided adisplay apparatus comprising: a memory configured to store one or moreinstructions; and a processor configured to execute the one or moreinstructions to: receive eye tracking information on an eye of a user ina captured image; identify whether the eye tracking is stable orunstable based on the eye tracking information; output a virtual objectin a two-dimensional (2D) image based on the eye tracking beingunstable; and output the virtual object in a three-dimensional (3D)image based on the eye tracking being stable.

According to another aspect of the disclosure, there is provided adisplay method comprising: receiving eye tracking information on an eyeof a user in an image; identifying whether the eye tracking is stable orunstable based on the eye tracking information; outputting a virtualobject in a two-dimensional (2D) image based on the eye tracking beingunstable; and outputting the virtual object in a three-dimensional (3D)image based on the eye tracking being stable.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will be more apparent by describingcertain example embodiments with reference to the accompanying drawings,in which:

FIG. 1A illustrates a head-up display (HUD) device according to anexample embodiment;

FIG. 1B illustrates a path of light in relation to an HUD deviceaccording to an example embodiment;

FIG. 2 illustrates a structure of a display device according to anexample embodiment;

FIG. 3 illustrates three-dimensional (3D) augmented reality (AR)according to an example embodiment;

FIG. 4 illustrates a process of generating an HUD image according to anexample embodiment;

FIG. 5 illustrates eye tracking statuses according to an exampleembodiment;

FIG. 6 illustrates eye movements in a viewing space according to anexample embodiment;

FIG. 7 illustrates a process of switching a rendering mode according toan example embodiment;

FIG. 8 illustrates buffer viewpoints and buffer source images forswitching a rendering mode according to an example embodiment;

FIG. 9 illustrates images in a 3D rendering mode according to an exampleembodiment;

FIG. 10 illustrates images in a two-dimensional (2D) rendering modeaccording to an example embodiment;

FIGS. 11 and 12 illustrate a process of tracking eyes using a trackingregion according to an example embodiment;

FIG. 13 illustrates a process of generating an HUD image based on eyetracking according to an example embodiment;

FIG. 14 illustrates a method of controlling an HUD considering an eyetracking status according to an example embodiment;

FIG. 15 illustrates a configuration of an HUD control apparatusaccording to an example embodiment; and

FIG. 16 illustrates a configuration of an electronic device according toan example embodiment.

DETAILED DESCRIPTION

Hereinafter, example embodiments will be described in detail withreference to the accompanying drawings. However, various alterations andmodifications may be made to the example embodiments. Here, the exampleembodiments are not construed as limited to the disclosure. The exampleembodiments should be understood to include all changes, equivalents,and replacements within the idea and the technical scope of thedisclosure.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not to be limiting of the exampleembodiments. The singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms“comprises/comprising” and/or “includes/including” when used herein,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components and/or groups thereof.

Unless otherwise defined, all terms including technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which example embodiments belong. It willbe further understood that terms, such as those defined in commonly-useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

When describing the example embodiments with reference to theaccompanying drawings, like reference numerals refer to like constituentelements and a repeated description related thereto will be omitted. Inthe description of example embodiments, detailed description ofwell-known related structures or functions will be omitted when it isdeemed that such description will cause ambiguous interpretation of thedisclosure.

Also, in the description of the components, terms such as first, second,A, B, (a), (b) or the like may be used herein when describing componentsof the disclosure. These terms are used only for the purpose ofdiscriminating one constituent element from another constituent element,and the nature, the sequences, or the orders of the constituent elementsare not limited by the terms. When one constituent element is describedas being “connected”, “coupled”, or “attached” to another constituentelement, it should be understood that one constituent element can beconnected or attached directly to another constituent element, and anintervening constituent element can also be “connected”, “coupled”, or“attached” to the constituent elements.

The same name may be used to describe an element included in the exampleembodiments described above and an element having a common function.Unless otherwise mentioned, the descriptions on the example embodimentsmay be applicable to the following example embodiments and thus,duplicated descriptions will be omitted for conciseness.

FIG. 1A illustrates a head-up display (HUD) device according to anexample embodiment. Referring to FIG. 1A, an HUD device 100 includes anHUD control apparatus 110, a display device 120, an eye tracking camera130, and a translucent optical device 140. The HUD device 100 may bemounted on a vehicle (for example, a car or an airplane) to provide anHUD image to a user (for example, a driver, a pilot, and the like). TheHUD device 100 may provide the HUD image using augmented reality (AR).For example, contents provided through an AR HUD may include dashboardinformation, navigation information, a lane indicator, a constructionindicator, an accident indicator, a pedestrian detection indicator, andthe like. AR may be applied to an HUD, a transmissive head-mounteddisplay (HMD), and the like. Hereinafter, an HUD will be described.However, the following description may also apply to an HMD or otherdisplay devices.

The display device 120 may include a light source, a display panel, athree-dimensional (3D) optical layer, and an optical element. Theoptical element may include a cata-dioptric system. Light correspondingto an HUD image may be provided by the display panel and the lightsource of the display device 120, and the cata-dioptric system mayreflect the light corresponding to the HUD image toward the translucentoptical device 140. In this case, the cata-dioptric system may refractthe light corresponding to the HUD image to enlarge the HUD image. Alight-emitting diode (LED) or a laser may be used as the light source.

A virtual screen 150 may be formed by the light corresponding to the HUDimage output by the display device 120. A portion of the light output bythe display device 120 may be reflected by the translucent opticaldevice 140 positioned in front of the user and viewable by the user. Thetranslucent optical device 140 may be a windshield of the car orairplane, or a combiner provided separately from the windshield for thepurpose of reflecting an HUD image. The user views light passing throughthe front of the translucent optical device 140, and a portion of thelight reflected by the translucent optical device 140 among the lightradiated by the display device 120 at the same time. Thus, a real objectand a virtual object may overlap each other and be provided to the useras AR content. For example, the real object may be an object in thesurrounding environment visible through the translucent optical device140.

The display device 120 may display the virtual object at a positioncorresponding to the real object. For example, traveling directioninformation of the vehicle, lane information, hazard information, andthe like may be displayed through the HUD as virtual objects atpositions corresponding to real objects. A position on the background orthe surrounding environment at which a virtual object is to be displayedmay be referred to as a target position. The HUD control apparatus 110may display the virtual object at the target position using atransformation relationship between a coordinate system of the eyetracking camera 130 and a coordinate system of the virtual screen 150,3D information on the background, and eye position information.

For example, the 3D information on the background may be obtainedthrough a camera or a 3D sensor provided to face ahead of the vehicle.Eye positions of the user may be obtained through the eye trackingcamera 130 provided to face the user. The eye tracking camera 130 maycapture the user and generate a user image including the user (forexample, the face of the user), and the HUD control apparatus 110 mayobtain the eye positions by performing eye tracking on the user image.The HUD control apparatus 110 may generate the HUD image for displayingthe virtual object at intersecting points where lines connecting the eyepositions of the user and the target position intersect with the virtualscreen 150, and the display device 120 may represent the virtual objectat the target position by displaying the HUD image.

The display device 120 may provide a 3D image through the 3D opticallayer. The HUD control apparatus 110 may generate a first source image(for example, a left image) for a first viewpoint (for example, the lefteye) and a second source image (for example, a right image) for a secondviewpoint (for example, the right eye), and render the HUD image basedon the eye positions tracked by the eye tracking camera 130, the firstsource image and the second source image. Here, a viewpoint maycorrespond to a viewing position (for example, a position of an eye of aviewer). An operation of rendering the HUD image may include determiningpixel values of the HUD image so that the first source image may beviewed at the first viewpoint and the second source image may be viewedat the second viewpoint. Hereinafter, an example of using two viewpointsfor a 3D image will be described. However, embodiments of the disclosureare not limited to the description below, and as such, according toanother example embodiment, two or more viewpoints may be used for alight field.

The display device 120 may display the HUD image generated as describedabove. The display device 120 may display the HUD image on the displaypanel. The HUD image may pass through the 3D optical layer and beprovided to the user. In this case, different images corresponding tothe first source image and the second source image may be provided toboth eyes of the user. For each of the first source image and the secondsource image, the HUD control apparatus 110 may render the HUD image sothat the virtual object may be displayed at the intersecting point wherea line connecting each eye position of the user and the target positionintersects with the virtual screen 150.

FIG. 1B illustrates a path of light in relation to an HUD deviceaccording to an example embodiment. Referring to FIG. 1B, an HUD device101 includes a display device 161 and mirrors 162 and 163. The displaydevice 161 may correspond to the display device 120 of FIG. 1A. Thedisplay device 161 may include a display panel and a light source andprovide light corresponding to an HUD image through the display paneland the light source. For example, the light source may include abacklight unit (BLU).

Light corresponding to the HUD image output by the display device 161may be reflected by the mirrors 162 and 163 and projected onto awindshield 170. At least one of the mirrors 162 and 163 may correspondto an aspheric surface mirror, and adjust a path of the lightcorresponding to the HUD image to enlarge the HUD image. A user may viewa virtual image corresponding to the HUD image on a virtual screen 180through light reflected by the windshield 170 toward an eye box 190.

In this way, the HUD system 100 may display information on the virtualscreen 180 provided in front of the user through projection. In order toprovide AR information through an HUD, the virtual screen 180 on whichthe HUD image is viewed may be implemented with a wide field of view(FOV). If the size of an image to be represented is not large enough orthe FOV is not wide enough, it may be difficult to represent informationon an object or background in front of a vehicle using AR.

The display device 161 and the mirrors 162 and 163 may be mounted in adashboard of the vehicle. The display device 161 and the mirrors 162 and163 may be designed to provide an FOV wide enough to implement AR. Forexample, the BLU of the display device 161 may optimize an output angleof the light output from the LED using a secondary lens array, andcompensate for a shortfall in the output angle using a side reflector.In this case, the diffusion angles of a diffuser plate and a polarizeddiffuser plate may be maintained at small values, so that a decrease inthe efficiency of the BLU may be prevented. Accordingly, it is possibleto achieve compact BLU volume, wide FOV, uniformity, improved sidebrightness, and the like.

FIG. 2 illustrates a structure of a display device according to anexample embodiment. Referring to FIG. 2, a display device 200 includes alight source 210, a diffuser 220, a display panel 230, and a 3D opticallayer 240. The light source 210 may correspond to a BLU. According to anexample embodiment, the light source may include a white LED, ared/green/blue (RGB) LED, or an RGB laser. If an aspheric mirror is usedas an enlarging and reflecting mirror, any of the white LED, the RGBLED, and the RGB laser may be used. However, if a holographic mirror isused, the RGB LED or the RGB laser may be used depending on recordingcharacteristics. The diffuser 220 may be implemented in the form of afilm, and light uniformity between the light source 210 and the displaypanel 230 may be provided through the diffuser 220. According to anexample embodiment, the diffuser 220 may be formed directly on thedisplay panel 230. According to another example embodiment, the diffuser220 may be spaced apart from the display panel 230.

The display panel 230 may include a liquid crystal display (LCD) panel,or a spatial light modulator (SLM) such as a digital light processor(DLP) and liquid crystal on silicon (LCoS). The 3D optical layer 240 maybe any one of a parallax barrier, a lenticular lens, and a directionalbacklight unit. The display panel 230 may display an HUD image, and the3D optical layer 240 may control a path of the light corresponding tothe HUD image. For example, the 3D optical layer 240 may givedirectivity to the light corresponding to the HUD image so that imagesof different viewpoints may be provided to both eyes of the user.

FIG. 3 illustrates 3D AR according to an example embodiment. Referringto FIG. 3, a virtual object 335 is displayed at an intersecting pointwhere a line connecting an eye position 310 of the user and a targetposition 350 intersects with a virtual screen 330. The eye position 310may be tracked through an eye tracking camera 320. In this case, a scene331 may be viewed at the eye position 310. The scene 331 includes thevirtual object 335 and a real object 340. The virtual object 335 may beaccurately displayed at the target position 350 through a relationshipbetween a coordinate system of the eye tracking camera 320 and acoordinate system of the virtual screen 330, 3D information on thebackground, and information on the eye position 310.

This process may be performed for each of the eyes of the user. Forexample, a first source image (for example, a left image) may begenerated so that the virtual object 335 may be displayed at anintersecting point where a line connecting a first viewpoint (forexample, the left eye) and the target position 350 intersects with thevirtual screen 330, and a second source image (for example, a rightimage) may be generated so that the virtual object 335 may be displayedat an intersecting point where a line connecting a second viewpoint (forexample, the right eye) and the target position 350 intersects with thevirtual screen 330. Thereafter, the scene 331 may be implemented as a 3DAR HUD by rendering the HUD image based on the first source image andthe second source image.

According to an example embodiment, the 3D HUD may represent the virtualobject 335 at various depths in response to a change in the position ofthe user, and as such, the 3D HUD may more accurately display thevirtual object 335 at the target position 350 than a 2D HUD. However, inorder to stably provide such a 3D HUD, continuous tracking of the eyeposition 310 may be necessary and the virtual object 335 may bedisplayed on the virtual screen 330 based on the tracked eye position310.

In an example scenario, the eye position 310 may not be tracked due toan environmental factor, such as low illuminance or because the eye iscovered. Moreover, an appropriate HUD image corresponding to the currenteye position 310 may not be generated due to a systemic factor such as asystem delay. In this example, a deterioration in the quality of the 3DHUD, such as crosstalk observed in the image as an image for the lefteye is provided to the right eye, may occur. In this case, drivinginformation may be stably provided by providing a 2D HUD instead of thelow-quality 3D HUD. According to example embodiments, 2D rendering or 3Drendering may be selectively performed based on a current circumstanceassociated with eye tracking, whereby the HUD stability may be improved.

FIG. 4 illustrates a process of generating an HUD image according to anexample embodiment. Operations 410 to 440 described below may beperformed on a current frame of a user image. Referring to FIG. 4, inoperation 410, an HUD control apparatus performs eye tracking. Forexample, the HUD control apparatus may generate a user image using aneye tracking camera and perform eye tracking on the user image. The HUDcontrol apparatus may generate an eye tracking result while performingeye tracking. If eye tracking is successful, the eye tracking result mayinclude eye coordinates. If eye tracking fails, the eye tracking resultmay not include eye coordinates. Instead, the eye tracking results mayinclude information indicating that eye tracking has failed. The eyecoordinates may include 3D coordinate values of each of the eyes.

In operation 420, the HUD control apparatus determines an eye trackingstatus. For example, the HUD control apparatus may classify the eyetracking status as one of a stable status and an unstable status basedon whether the eye tracking result complies with a 3D renderingcondition. Here, the 3D rendering condition may be defined based on thepresence of eye coordinates and a rate of change of the eye coordinates.As described above, in order to maintain the quality of a 3D HUD image,eye coordinates must be identified, and system performance for trackinga change in the eye coordinates is required.

For example, if eye coordinates are absent, or if there are eyecoordinates but the eye coordinates change so severely that therendering performance of the system is incapable of coping with thechange, crosstalk is likely to be observed in a 3D HUD image.Accordingly, in a first state in which the eye tracking result includesthe eye coordinates and a speed of position change with respect to theeye coordinates is less than a threshold, the eye tracking status may beclassified as the stable state. In this case, the threshold maycorrespond to a system processing rate. In addition, in a second statein which the eye tracking result includes the eye coordinates and thespeed of position change with respect to the eye coordinates is greaterthan the threshold, or in a third state in which the eye tracking resultdoes not include the eye coordinates, the eye tracking status may beclassified as the unstable status.

In operation 430, the HUD control apparatus determines a rendering mode.Here, the determined rendering mode may be used to render an HUD imagecorresponding to a current frame of the user image. The rendering modemay include a 2D rendering mode and a 3D rendering mode. The HUD controlapparatus may determine the rendering mode for an HUD image to be one ofthe 2D rendering mode and the 3D rendering mode based on the eyetracking status. For example, if the eye tracking status is classifiedas the stable status, the HUD control apparatus may determine therendering mode to be the 3D rendering mode. Conversely, if the eyetracking status is classified as the unstable status, the HUD controlapparatus may determine the rendering mode to be the 2D rendering mode.

The HUD control apparatus may render the HUD image so that the same HUDimage is provided to both eyes of the user in the 2D rendering mode, ormay render the HUD image so that different images are provided to botheyes of the user in the 3D rendering mode. For example, the HUD controlapparatus may generate a first source image (for example, a left image)for a first viewpoint (for example, the left eye) and a second sourceimage (for example, a right image) for a second viewpoint (for example,the right eye), and render the HUD image so that the first source imagemay be provided to the first viewpoint and the second source image maybe provided to the second viewpoint. If the rendering mode is determinedto be the 2D rendering mode, the HUD control apparatus may render theHUD image by setting the first viewpoint and the second viewpointequally as a single viewpoint. Conversely, if the rendering mode isdetermined to be the 3D rendering mode, the HUD control apparatus mayrender the HUD image by setting the first viewpoint and the secondviewpoint as different viewpoints.

The 2D rendering mode may include a tracking 2D rendering mode and afixed 2D rendering mode. As described above, in the second state inwhich the eye tracking result includes the eye coordinates and the speedof position change with respect to the eye coordinates is greater thanthe threshold, or in the third state in which the eye tracking resultdoes not include the eye coordinates, the eye tracking status may beclassified as the unstable status. In the case of the second state,since the eye coordinates are present, the tracking 2D rendering modemay be performed using the eye coordinates. For example, if the eyetracking result includes current eye coordinates of both eyes and aspeed of position change with respect to the current eye coordinates isgreater than the threshold, the first viewpoint of the first sourceimage and the second viewpoint of the second source image may setequally as a center viewpoint of the current eye coordinates. On theother hand, in the case of the third state, since eye coordinates areabsent, the fixed 2D rendering mode may be performed using eyecoordinates previously obtained. For example, if the eye tracking resultdoes not include the current eye coordinates, the first viewpoint andthe second viewpoint may be set equally as a center viewpoint ofprevious eye coordinates recently used.

In operation 440, the HUD control apparatus renders the HUD image in thedetermined rendering mode. The HUD image may be displayed by a displaydevice and provided to the user through a 3D optical layer. If 3Drendering is performed, the HUD image may traverse through the 3Doptical layer such that images of different viewpoints may be providedto both eyes of the user. Even if 2D rendering is performed, the HUDimage may be provided to the user through the 3D optical layer. However,in this case, unlike 3D rendering, an image of the same viewpoint may beprovided to both eyes of the user. After operation 440 is performed onthe current frame as described above, operations 410 to 440 may beperformed on a subsequent frame. This process may be performed for eachframe of the user image.

FIG. 5 illustrates eye tracking status according to an exampleembodiment. Referring to FIG. 5, eye coordinates at positions markedwith “X” may be obtained for each frame in a user image 510 byperforming eye tracking on the user image 510. In addition, since theeye coordinates for each frame do not change greatly, an eye trackingstatus of the user image 510 may be classified as a stable status.

For a user image 520 as well, eye coordinates at positions marked with“X” may be obtained for each frame through eye tracking. However, sincethe eye coordinates for each frame of the user image 520 change greatly,an eye tracking status of the user image 520 may be classified as anunstable status. For example, when a vehicle drives over a speed bump,drives on an uneven road, or takes a sharp curve, the eye positions mayquickly change as shown in the user image 520.

A user image 530 does not have “X” marks corresponding to eye positionseach of the frames illustrating a case in which eye tracking fails. Forexample, eye tracking may fail as in the user image 530 due to anenvironmental factor such as low illuminance or occlusion.

If the eye tracking status is classified as a stable status as in theuser image 510, the HUD image may be rendered through a 3D renderingmode. If the eye tracking status is classified as an unstable status asin the user images 520 and 530, the HUD image may be rendered through a2D rendering mode. If eye coordinates are present as in the user image520, a tracking 2D rendering mode may be performed. If eye coordinatesare absent as in the user image 530, a fixed 2D rendering mode may beperformed.

FIG. 6 illustrates eye movements in a viewing space according to anexample embodiment. Referring to FIG. 6, a viewing space 600 includes afirst viewing space S₁ in which a first source image is viewed and asecond viewing space S₂ in which a second source image is viewed. An eyeposition 611 is a position of a first viewpoint (for example, the lefteye) at a time t₁, and an eye position 621 is a position of a secondviewpoint (for example, the right eye) at the time t₁. A differencebetween the time t₁ and a time t₂ may correspond to a time differencebetween two consecutive frames. A user may view the first source imagethrough the first viewpoint of the eye position 611 and view the secondsource image through the second viewpoint of the eye position 621. Thefirst viewing space S₁ and the second viewing space S₂ may be dividedthrough a borderline 630. An HUD control apparatus may adjust theborderline 630 in response to changes in the eye positions 611 and 621,so that the eye position 611 may stay in the first viewing space S₁ andthe eye position 621 may stay in the second viewing space S₂.

The eye position 611 is the position of the first viewpoint at the timet₁, and an eye position 612 is a position of the first viewpoint at thetime t₂. Further, the eye position 621 is the position of the secondviewpoint at the time t₁, and an eye position 622 is a position of thesecond viewpoint at the time t₂. Thus, a speed of change (or speed ofmovement) of the eye positions 611 and 621 (or the eye coordinates) maybe defined as V_(e). In addition, a speed of adjustment (or speed ofmovement) of the borderline 630 may be defined as V_(t). V_(e) and V_(t)may correspond to a variation of the eye positions 611 and 621 and avariation of the borderline 630 during the time difference between thetwo consecutive frames. Since system processing such as updating an HUDimage is required to adjust the borderline 630, the maximum value ofV_(t) may be limited by the system processing rate. If V_(e) is greaterthan the maximum value of V_(t), the eye position 612 of the firstviewpoint may be in the second viewing space S₂, and the eye position622 of the second viewpoint may be in the first viewing space S₁, forexample, as shown in FIG. 6. Accordingly, crosstalk may be observed.

A threshold may be set based on the system processing rate. Thethreshold may be a speed of the borderline 630 that is adjustable to themaximum based on the system processing rate. For example, the thresholdmay be set to 240 millimeters per second (mm/s). In this case, if aspeed of change of the eye positions 611 and 621 (or the eyecoordinates) is greater than the threshold in a frame of a user image,an eye tracking status for the frame may be determined to be an unstablestatus. Accordingly, an HUD image corresponding to the frame may berendered through a 2D rendering mode. In detail, since eye coordinatesare present, a tracking 2D rendering mode may be used.

FIG. 7 illustrates a process of switching a rendering mode according toan example embodiment. For example, the switching of the rendering modemay include switching from a 2D rendering mode to a 3D rendering modeand switching from a 3D rendering mode to a 2D rendering mode. Inanother example, the switching of the rendering mode may includeswitching from one of a tracking 2D rendering mode, a fixed 2D renderingmode, and a 3D rendering mode to another of the tracking 2D renderingmode, the fixed 2D rendering mode, and the 3D rendering mod. While therendering mode is switched, a change in viewpoint may occur in an HUDimage and cause a user to feel uncomfortable when viewing the HUD image.According to example embodiments, to reduce such discomfort, switchingof the rendering mode may be performed for a predetermined time.

Referring to FIG. 7, in operation 710, the HUD control apparatusdetermines whether to switch the rendering mode. For example, inoperation 710, after the rendering mode is determined in operation 430of FIG. 4, it is determined as to whether to switch the rendering modebased on the determination of operation 430. For example, if therendering mode is determined to be a 3D rendering mode in operation 430for iteration on a previous frame and the rendering mode is determinedto be a 2D rendering mode in operation 430 for iteration on a currentframe, the rendering mode is to be switched. In this case, operation 720may be performed.

In operation 720, the HUD control apparatus performs a switchingoperation during a buffer time. The buffer time may correspond to aplurality of frames. For example, if the frame rate of the HUD image is60 frames per second (fps), the buffer time may correspond to 1 sec=60frames. If the rendering mode is to be switched from the 2D renderingmode to the 3D rendering mode, the HUD control apparatus may render theHUD image while gradually changing a first viewpoint and a secondviewpoint to a single viewpoint used in the 2D rendering mode over thebuffer time. The switching operation will be described further withreference to FIG. 8.

If the rendering mode is determined to be a 3D rendering mode inoperation 430 for iteration on a previous frame and the rendering modeis maintained to be the 3D rendering mode in operation 430 for iterationon a current frame, the rendering mode is not to be switched. In thiscase, the HUD image corresponding to the current frame may be renderedin the 3D rendering mode through operation 440 of FIG. 4.

FIG. 8 illustrates buffer viewpoints and buffer source images forswitching a rendering mode according to an example embodiment. FIG. 8shows an example of switching a rendering mode from a 3D rendering modeto a 2D rendering mode. However, the disclosure is not limited thereto,and the example of FIG. 8 and the following description may also applyto another type of switching process according to another exampleembodiment. Referring to FIG. 8, a first viewpoint 820 (for example, theleft eye), a second viewpoint 830 (for example, the right eye), and acenter viewpoint 840 are shown in a viewing space 810. The centerviewpoint 840 may be in the middle of the first viewpoint 820 and thesecond viewpoint 830. Buffer viewpoints may be between the firstviewpoint 820 and the center viewpoint 840 and between the secondviewpoint 830 and the center viewpoint 840. The number of bufferviewpoints may correspond to a buffer time. For example, if the buffertime corresponds to 60 frames, 60 buffer viewpoints may be between thefirst viewpoint 820 and the center viewpoint 840, and 60 bufferviewpoints may be between the second viewpoint 830 and the centerviewpoint 840.

A first source image 825 may correspond to the first viewpoint 820, asecond source image 835 may correspond to the second viewpoint 830, anda center source image 845 may correspond to the center viewpoint 840. Inaddition, there may be buffer source images corresponding to the bufferviewpoints. In a 3D rendering mode, a 3D HUD may be provided byrendering the HUD image based on the first source image 825 and thesecond source image 835. Further, in a 2D rendering mode, a 2D HUD maybe provided by rendering the HUD image based on the center source image845. If the rendering mode is switched from the 3D rendering mode to the2D rendering mode, a 3D HUD image may be rendered based on the firstsource image 825 and the second source image 835 at a time t₁, bufferHUD images may be rendered based on buffer source images from a time t₂to a time t_(B-1), and a 2D HUD image may be rendered based on thecenter source image 845 at a time t_(B). B may correspond to a buffertime.

The buffer source images may be generated based on an interpolationoperation based on the first source image 825, the second source image835, and the center source image 845. For example, center source imagescorresponding to the buffer viewpoints between the first viewpoint 820and the center viewpoint 840 may be generated through an interpolationoperation using the first source image 825 and the center source image845, and center source images corresponding to the buffer viewpointsbetween the second viewpoint 830 and the center viewpoint 840 may begenerated through an interpolation operation using the second sourceimage 835 and the center source image 845.

FIG. 9 illustrates images in a 3D rendering mode according to an exampleembodiment. Referring to FIG. 9, a first source image 910 includes avirtual object 915 at a position corresponding to a first viewpoint, anda second source image 920 includes a virtual object 925 at a positioncorresponding to a second viewpoint. In FIG. 9, the first viewpoint andthe second viewpoint are different viewpoints. The first source image910 is generated for displaying the virtual object 915 at anintersecting point where a line connecting the first viewpoint and atarget position intersects with a virtual screen, and the second sourceimage 920 is generated for displaying the virtual object 925 at anintersecting point where a line connecting the second viewpoint and thetarget position intersects with the virtual screen. An HUD image 930 maybe generated through 3D rendering based on the first source image 910and the second source image 920, such that a user viewing the HUD image930 may experience the virtual object 915 in 3D.

FIG. 10 illustrates images in a 2D rendering mode according to anexample embodiment. Unlike the example of FIG. 9, the first viewpointand the second viewpoint in FIG. 10 correspond to the same viewpoint.Thus, a virtual object 1015 of a first source image 1010 and a virtualobject 1025 of a second source image 1020 are located at the sameposition. An HUD image 1030 may be generated through 2D rendering basedon the first source image 1010 and the second source image 1020.

For example, the first source image 910 and the second source image 920of FIG. 9 may correspond to the first source image 825 and the secondsource image 835 of FIG. 8. Further, the first source image 1010 and thesecond source image 1020 of FIG. 10 may correspond to the center sourceimage 845 of FIG. 8. In addition, the buffer source images of FIG. 8 maybe generated through an interpolation operation based on the firstsource image 910, the second source image 920, and the first sourceimage 1010 (or the second source image 1020).

FIGS. 11 and 12 illustrate a process of tracking eyes using a trackingregion according to an example embodiment. The process of tracking theeyes may include using a tracking region described below. Referring toFIG. 11, operations 1110 and 1111 are operations performed on a firstframe F1 of a user image. In operation 1110, an HUD control apparatusperforms eye detection on an entire region of an image of the firstframe F1. For example, the HUD control apparatus may determine adetection region corresponding to eyes by scanning the entire image. Inoperation 1111, the HUD control apparatus determines a tracking region.The HUD control apparatus may determine the tracking region based on thedetection region. For example, the size of the tracking region may bedetermined based on the size of the detection region, and the positionof the tracking region may be determined to include the detection regionat the center thereof.

Next, operations 1121 to 1123 are operations performed on a second frameF2 of the user image. In operation 1121, the HUD control apparatusperforms eye tracking based on the tracking region. For example, the HUDcontrol apparatus may detect the eyes within the tracking region byscanning the tracking region, rather than scanning an entire region ofan image of the second frame F2. Such eye detection using the trackingregion may be referred to as eye tracking. In operation 1122, the HUDcontrol apparatus determines whether there are eyes in the trackingregion. If the eyes are in the tracking region, the HUD controlapparatus updates the tracking region, in operation 1123. In the samemanner as the tracking region is determined based on the detectionregion of the first frame F1, the tracking region may be updated basedon a detection region of the second frame F2.

Next, operations 1130 to 1133 are operations performed on a third frameF3 of the user image. In response to the determination of operation 1122that there are no eyes in the tracking region, the HUD control apparatusterminates a tracking mode and returns to a detection mode to performeye detection in an entire region of an image of the third frame F3, inoperation 1130. If the tracking region is updated in operation 1123 inresponse to the determination of operation 1122 that there are eyes inthe tracking region, the HUD control apparatus performs eye trackingbased on the updated tracking region, in operation 1131. The details ofoperations 1131 to 1133 are the same as those of operations 1121 to1123. As described above, if there are eyes in a tracking region, thetracking region may be updated and a tracking mode may be maintained.However, if there are no eyes in the tracking region, a detection modemay be activated again to scan the entire image.

Referring to FIG. 12, frames F1, F2, and F3 of a user image are shown.The HUD control apparatus determines a detection region in the frame F1,and determines a first tracking region 1210 based on the detectionregion. The eyes in the frame F2 may be at an upper right position thanthe eyes in the frame F1. The HUD control apparatus detects the eyesfrom a first tracking region 1210 in the frame F2. Since the eyes aredetected in the first tracking region 1210, the HUD control apparatusupdates a tracking region based on a detection region within the firsttracking region 1210 in the frame F2. Accordingly, a second trackingregion 1220 is determined. In the same manner as in the frame F2, theeyes may be detected from the second tracking region 1220 in the frameF3, and a third tracking region 1230 may be determined by updating thetracking region. In this way, in response to the determination that theeyes are included in each tracking region, the HUD control apparatus maytrack the eyes without scanning the entire image.

FIG. 13 illustrates a process of generating an HUD image based on eyetracking according to an example embodiment. Referring to FIG. 13, inoperation 1310, an HUD control apparatus determines an eye trackingstatus. In operation 1320, the HUD control apparatus determines whetherthe eye tracking status is a stable status. According to the exampleembodiments described with reference to FIGS. 11 and 12, eye trackingmay be performed using a tracking region.

If the eye tracking status corresponds to the stable status, the HUDcontrol apparatus performs 3D rendering, in operation 1330. If the eyetracking status does not correspond to the stable status, the HUDcontrol apparatus performs 2D rendering, in operation 1340. For example,if the eye positions are determined in the tracking region through eyetracking, but a speed of change in the eye positions is faster than athreshold, tracking 2D rendering may be performed, in operation 1341.When the eye positions are not determined in the tracking region througheye tracking, fixed 2D rendering may be performed, in operation 1342. Inthis case, a tracking mode for eye tracking may be canceled, and adetection mode may be activated again.

While 2D rendering is performed through operation 1340, the HUD controlapparatus checks whether the status changes, in operation 1350. Forexample, a case in which the status changes may include a case in whichthe speed of change in the eye positions is reduced below the thresholdwhile tracking 2D rendering is performed, and a case in which the speedof change in the eye positions is less than the threshold although theeyes are detected outside the tracking region while fixed 2D renderingis performed. If the status changes, the HUD control apparatus switchesthe rendering mode from a 2D rendering mode to a 3D rendering modeduring a buffer time, in operation 1360. Then, in operation 1330, theHUD control apparatus performs 3D rendering.

Similarly, while 3D rendering is performed through operation 1330, theHUD control apparatus checks whether the status changes, in operation1370. For example, a case in which the status changes may include a casein which the speed of change in the eye positions exceeds the threshold,and a case in which the eyes are not detected in a tracking region. Ifthe status changes, the HUD control apparatus switches the renderingmode from the 3D rendering mode to the 2D rendering mode during a buffertime, in operation 1380. Then, in operation 1340, the HUD controlapparatus performs 2D rendering.

FIG. 14 illustrates a method of controlling an HUD considering an eyetracking status according to an example embodiment, and furtherdescribes the example of FIG. 4. Referring to FIG. 14, in operation1410, an HUD control apparatus generates an eye tracking result byperforming eye tracking on a user image. In operation 1420, the HUDcontrol apparatus determines an eye tracking status related to a changein eye positions based on the eye tracking result. In operation 1430,the HUD control apparatus determines a rendering mode for an HUD imageto be one of a 2D rendering mode and a 3D rendering mode based on theeye tracking status. In operation 1440, the HUD control apparatusrenders the HUD image in the determined rendering mode. In addition, thedescription provided with reference to FIGS. 1A to 13, 15, and 16 mayapply to the method of controlling an HUD, and thus, a detaileddescription will be omitted for conciseness.

FIG. 15 illustrates a configuration of an HUD control apparatusaccording to an example embodiment. Referring to FIG. 15, an HUD controlapparatus 1500 includes a processor 1510 and a memory 1520. The memory1520 is connected to the processor 1510, and may store instructionsexecutable by the processor 1510, data to be computed by the processor1510, or data processed by the processor 1510. The memory 1520 mayinclude a non-transitory computer-readable medium (for example, ahigh-speed random access memory) and/or a non-volatile computer-readablemedium (for example, a disk storage device, a flash memory device, oranother non-volatile solid-state memory device). However, the disclosureis not limited thereto, and according to another example embodiment, thememory 1520 may be storage devices configured to store data, informationand/or instructions.

The processor 1510 may execute instructions to perform the operationsdescribed with reference to FIGS. 1A to 14, and 16. For example, theprocessor 1510 may generate an eye tracking result by performing eyetracking on the user image, determine an eye tracking status related toa change in eye positions based on the eye tracking result, determine arendering mode for an HUD image to be one of a 2D rendering mode and a3D rendering mode based on the eye tracking status, and render the HUDimage in the determined rendering mode. In addition, the descriptionprovided with reference to FIGS. 1A to 14, and 16 may apply to the HUDcontrol apparatus 1500, and thus, a detailed description will be omittedfor conciseness.

FIG. 16 illustrates a configuration of an electronic device according toan example embodiment. Referring to FIG. 16, an electronic device 1600may acquire a user image, track eyes from the acquired user image, andprovide an AR HUD image based on an eye tracking status. The electronicdevice 1600 may structurally and/or functionally include the HUD device100 of FIG. 1A, the HUD control apparatus 110 of FIG. 1A and/or the HUDcontrol apparatus 1500 of FIG. 15. For instance, the HUD device 100 ofFIG. 1A, the HUD control apparatus 110 of FIG. 1A and/or the HUD controlapparatus 1500 of FIG. 15 may be implemented as the electronic device1600.

The electronic device 1600 may include a processor 1610, a memory 1620,a camera 1630, a storage device 1640, an input device 1650, an outputdevice 1660, and a network interface 1670. However, the disclosure isnot limited thereto, and as such, according to another exampleembodiment, the electronic device 1600 may include other components orone or more of the components illustrated in FIG. 16 may be omitted fromthe electronic device 1600. The processor 1610, the memory 1620, thecamera 1630, the storage device 1640, the input device 1650, the outputdevice 1660, and the network interface 1670 may communicate with eachother through a communication bus 1680. For example, the electronicdevice 1600 may be implemented as part of a means of transportation suchas a car or an airplane.

The processor 1610 executes instructions or functions to be executed inthe electronic device 1600. For example, the processor 1610 may processthe instructions stored in the memory 1620 or the storage device 1640.The processor 1610 may perform the operations described through FIGS. 1Ato 15.

The memory 1620 stores a variety of data for providing an HUD image. Thememory 1620 may include a computer-readable storage medium or acomputer-readable storage device. The memory 1620 may store instructionsto be executed by the processor 1610 and may store related informationwhile software and/or an application is executed by the electronicdevice 1600.

The camera 1630 may capture a photo and/or a video. For example, thecamera 1630 may capture a user image including a user (for example, theface of the user). In detail, the camera 1630 may include the eyetracking camera 130 of FIG. 1A. The camera 1630 may provide a 3D imageincluding depth information related to objects.

The storage device 1640 includes a computer-readable storage medium orcomputer-readable storage device. The memory 1640 may store a variety ofdata for providing an HUD image. The storage device 1640 may store amore quantity of information than the memory 1620 for a long time. Forexample, the storage device 1640 may include a magnetic hard disk, anoptical disk, a flash memory, a floppy disk, or other non-volatilememories known in the art.

The input device 1650 may receive an input from the user in traditionalinput manners through a keyboard and a mouse, and in new input mannerssuch as a touch input, a voice input, and an image input. For example,the input device 1650 may include a keyboard, a mouse, a touch screen, amicrophone, or any other device that detects the input from the user andtransmits the detected input to the electronic device 1600.

The output device 1660 may provide an output of the electronic device1600 to the user through a visual, auditory, or haptic channel. Theoutput device 1660 may include, for example, a display, a touch screen,a speaker, a vibration generator, or any other device that provides theoutput to the user. In detail, the output device 1660 may include thedisplay device 120 of FIG. 1A. The network interface 1670 maycommunicate with an external device through a wired or wireless network.

The methods according to the above-described example embodiments may berecorded in non-transitory computer-readable media including programinstructions to implement various operations of the above-describedexample embodiments. The media may also include, alone or in combinationwith the program instructions, data files, data structures, and thelike. The program instructions recorded on the media may be thosespecially designed and constructed for the purposes of exampleembodiments, or they may be of the kind well-known and available tothose having skill in the computer software arts. Examples ofnon-transitory computer-readable media include magnetic media such ashard disks, floppy disks, and magnetic tape; optical media such asCD-ROM discs, DVDs, and/or Blue-ray discs; magneto-optical media such asoptical discs; and hardware devices that are specially configured tostore and perform program instructions, such as read-only memory (ROM),random access memory (RAM), flash memory (e.g., USB flash drives, memorycards, memory sticks, etc.), and the like. Examples of programinstructions include both machine code, such as produced by a compiler,and files containing higher-level code that may be executed by thecomputer using an interpreter. The devices may be configured to act assoftware modules in order to perform the operations of theabove-described examples, or vice versa.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independently or uniformlyinstruct or configure the processing device to operate as desired.Software and data may be embodied permanently or temporarily in any typeof machine, component, physical or virtual equipment, computer storagemedium or device, or in a propagated signal wave capable of providinginstructions or data to or being interpreted by the processing device.The software also may be distributed over network-coupled computersystems so that the software is stored and executed in a distributedfashion. The software and data may be stored by non-transitorycomputer-readable recording mediums.

A number of example embodiments have been described above. Nevertheless,it should be understood that various modifications may be made to theseexample embodiments. For example, suitable results may be achieved ifthe described techniques are performed in a different order and/or ifcomponents in a described system, architecture, device, or circuit arecombined in a different manner and/or replaced or supplemented by othercomponents or their equivalents. Accordingly, other implementations arewithin the scope of the following claims.

What is claimed is:
 1. A method of controlling a head-up display (HUD),the method comprising: performing eye tracking of an eye of a user in acaptured image; identifying an eye tracking status based on a result ofthe eye tracking; identifying a rendering mode for an HUD image to beone of a two-dimensional (2D) rendering mode and a three-dimensional(3D) rendering mode based on the eye tracking status; and rendering theHUD image in the identified rendering mode.
 2. The method of claim 1,wherein the identifying the eye tracking status comprises classifyingthe eye tracking status as one of a stable status and an unstable statusbased on whether eye coordinates are present in the result of the eyetracking or based on a rate of change of the eye coordinates.
 3. Themethod of claim 2, wherein the identifying the rendering mode comprises:identifying the rendering mode to be the 3D rendering mode based on theeye tracking status being classified as the stable status; andidentifying the rendering mode to be the 2D rendering mode based on theeye tracking status being classified as the unstable status.
 4. Themethod of claim 2, wherein the eye tracking status is classified as thestable status based on the eye coordinates being included in the resultof the eye tracking and a speed of change of the eye coordinates is lessthan a reference value.
 5. The method of claim 4, wherein the referencevalue corresponds to a system processing rate.
 6. The method of claim 2,wherein the eye tracking status is classified as the unstable statusbased on the eye coordinates being included in the result of the eyetracking and a speed of change of the eye coordinates is greater than areference value, or based on the eye coordinates not being included inthe result of the eye tracking.
 7. The method of claim 1, wherein theHUD image is rendered based on a first source image for a firstviewpoint and a second source image for a second viewpoint.
 8. Themethod of claim 7, wherein, based on the identified rendering mode beingthe 2D rendering, the rendering the HUD image comprises rendering theHUD image by setting the first viewpoint and the second viewpointequally as a single viewpoint.
 9. The method of claim 8, wherein therendering the HUD image comprises: setting, based on the result of theeye tracking including current eye coordinates of both eyes and a speedof change of the current eye coordinates being greater than a referencevalue, the first viewpoint and the second viewpoint equally as a centerviewpoint of the current eye coordinates; and setting, based on theresult of the eye tracking not including the current eye coordinates,the first viewpoint and the second viewpoint equally as a centerviewpoint of previous eye coordinates.
 10. The method of claim 7,wherein, based on the identified rendering mode being the 3D renderingmode, the rendering the HUD image comprises rendering the HUD image bysetting the first viewpoint and the second viewpoint as differentviewpoints.
 11. The method of claim 1, wherein based on the renderingmode being identified, the rendering mode is switched from the 3Drendering mode to the 2D rendering mode or from the 2D rendering mode tothe 3D rendering mode during a buffer time corresponding to a pluralityof frames.
 12. The method of claim 11, wherein the HUD image is renderedbased on a first source image for a first viewpoint and a second sourceimage for a second viewpoint, and wherein, based on the rendering modebeing switched from the 2D rendering mode to the 3D rendering mode, therendering the HUD image comprises rendering the HUD image whilegradually changing the first viewpoint and the second viewpoint to asingle viewpoint used in the 2D rendering mode over the buffer time. 13.A non-transitory computer-readable storage medium storing instructionsthat, when executed by a processor, cause the processor to perform themethod of claim
 1. 14. An apparatus for controlling a head-up display(HUD), the apparatus comprising: a memory configured to store one ormore instructions; and a processor configured to execute the one or moreinstructions to: perform eye tracking of an eye of a user in a capturedimage, identify an eye tracking status based on a result of the eyetracking, identify a rendering mode for an HUD image to be one of atwo-dimensional (2D) rendering mode and a three-dimensional (3D)rendering mode based on the eye tracking status, and render the HUDimage in the identified rendering mode.
 15. The apparatus of claim 14,wherein the processor is further configured to classify the eye trackingstatus as one of a stable status and an unstable status based on whethereye coordinates are present in the result of the eye tracking or basedon a rate of change of the eye coordinates.
 16. The apparatus of claim15, wherein the processor is further configured to: identify therendering mode to be the 3D rendering mode based on the eye trackingstatus being classified as the stable status; and identify the renderingmode to be the 2D rendering mode based on the eye tracking status beingclassified as the unstable status.
 17. The apparatus of claim 14,wherein, based on the rendering mode being identified, the renderingmode is switched from the 3D rendering mode to the 2D rendering mode orfrom the 2D rendering mode to the 3D rendering mode during a buffer timecorresponding to a plurality of frames.
 18. A head-up display (HUD)device comprising: an eye tracking camera configured to capture an imageincluding a user; a processor configured to perform eye tracking on thecaptured image, identify an eye tracking status based on a result of theeye tracking, identify a rendering mode for an HUD image to be one of atwo-dimensional (2D) rendering mode and a three-dimensional (3D)rendering mode based on the eye tracking status, and render the HUDimage in the identified rendering mode; and a display device configuredto provide the HUD image to the user using augmented reality (AR). 19.The HUD device of claim 18, wherein the processor is further configuredto classify the eye tracking status as one of a stable status and anunstable status based on whether eye coordinates are present in theresult of the eye tracking and based on a rate of change of the eyecoordinates.
 20. The HUD device of claim 18, wherein the processor isfurther configured to, based on the rendering mode being identified,switch the rendering mode from the 3D rendering mode to the 2D renderingmode or from the 2D rendering mode to the 3D rendering mode during abuffer time corresponding to a plurality of frames.
 21. A displayapparatus comprising: a memory configured to store one or moreinstructions; and a processor configured to execute the one or moreinstructions to: receive eye tracking information on an eye of a user ina captured image; identify whether the eye tracking is stable orunstable based on the eye tracking information; output a virtual objectin a two-dimensional (2D) image based on the eye tracking beingunstable; and output the virtual object in a three-dimensional (3D)image based on the eye tracking being stable.
 22. A display methodcomprising: receiving eye tracking information on an eye of a user in animage; identifying whether the eye tracking is stable or unstable basedon the eye tracking information; outputting a virtual object in atwo-dimensional (2D) image based on the eye tracking being unstable; andoutputting the virtual object in a three-dimensional (3D) image based onthe eye tracking being stable.